In a block-based predictive video coding model, an image block can be coded by generating a prediction, subtracting the prediction from the image block to generate the residue, transforming the residue, and quantizing the transformed residue, and finally transmitting the quantized residue coefficients. Since the quantization process can cause information loss, visible blocking artifacts may often be produced between adjacent coding blocks. In particular, the smooth region and the edge between adjacent coding blocks may appear discontinuous, which is highly undesirable.
In order to remove or reduce such artifacts, some filtering can be performed in order to smooth the transition between adjacent coding blocks which are likely to present some blocking artifacts. The filtering strength on block boundaries often depends on the prediction modes used in each coding block, since correlations exist between the coding modes used and the blocking artifact perceived at a given bitrate. In intra prediction, most of the current video encoding techniques (e.g., intra prediction in accordance with the International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group-4 (MPEG-4) Part 10 Advanced Video Coding (AVC) standard/International Telecommunication Union, Telecommunication Sector (ITU-T) H.264 Recommendation (hereinafter the “MPEG-4 AVC Standard”)) exploit local neighboring information to generate the prediction of the current block. Therefore, current deblocking filtering methods for intra predicted blocks are designed based on the knowledge of local information.
Recently, some non-local intra prediction approaches have been introduced and have achieved good coding efficiency. One such example includes displacement intra prediction (DIP). Another such example includes template matching prediction (TMP). These approaches try to exploit a region's self-similarity existing within the picture. Specifically, non-local intra prediction techniques can use non-local information within the decoded portion of the encoding picture to generate the prediction, which is different from the local intra prediction techniques (e.g., intra prediction in accordance with the MPEG-4 AVC Standard) that exploit only the local neighboring data. When such non-local prediction techniques are introduced in intra prediction, deblocking filtering methods and strategies designed so far may fail in properly filtering out blocky artifacts. Indeed, the filtering strategy defined for local intra prediction should be modified for more efficient deblocking filtering of other prediction approaches such as non-local prediction.
In the MPEG-4 AVC Standard, a filter is applied to each decoded macroblock in order to reduce blocking artifacts. The deblocking filter is applied on reconstructed picture data in both the encoder and decoder. The filter smoothes block transitions, improving the appearance of decoded frames. Filtering is applied to vertical or horizontal edges of 4×4 blocks in a macroblock (except for edges on the slice boundaries). The default deblocking filtering order is to filter edges in the luma component first, and then to filter edges in the chroma components. In each component, vertical edges are filtered before horizontal edges. Turning to FIGS. 1A and 1B, a 16×16 luma component of a macroblock and a 8×8 chroma component of the macroblock are indicated generally by the reference numerals 100 and 150, respectively. The edge filtering order in the macroblock is as follows: a; b; c; d; e; f; g; h; i; j; k; and l. Each filtering operation affects up to three samples on either side of the boundary. Turning to FIGS. 2A and 2B, samples adjacent to vertical and horizontal boundaries are indicated generally by the reference numerals 200 and 250, respectively. In particular, four samples are shown on either side of a vertical or horizontal boundary in adjacent blocks p and q (p0, p1, p2, p3 and q0, q1, q2, q3, respectively). In the MPEG-4 AVC Standard, the filter used for deblocking in a particular location depends on the boundary strength (BS), the gradient of image samples across the boundary, and the current quantization strength.
Turning to FIG. 3, an exemplary method for selecting a deblocking filtering boundary strength (bS) with respect to the MPEG-4 AVC Standard is indicated generally by the reference numeral 300. The method 300 includes a start block 305 that passes control to a function block 310. The function block 310 inputs p and q blocks and intra prediction modes, and passes control to a decision block 315. The decision block 315 determines whether the block p or the block q is intra coded. If so, then control is passed to a decision block 320. Otherwise, control is passed to a decision block 335.
The decision block 320 determines whether or not the block boundary is a macroblock boundary. If so, then control is passed to a function block 325. Otherwise, control is passed to a function block 330.
The function block 325 sets the boundary strength (bS) equal to four, and passes control to a function block 365.
The function block 365 outputs the boundary strength (bS), and passes control to an end block 399.
The function block 330 sets the boundary strength (bS) equal to three, and passes control to the function block 365.
The decision block 335 determines whether or not there is a coefficient coded in block p or block q. If so, then control is passed to a function block 340. Otherwise, control is passed to a decision block 345.
The function block 340 sets the boundary strength (bS) equal to two, and passes control to the function block 365.
The decision block 345 determines whether or not the block p and the block q have a different reference frame number of a different number of reference frames. If so, then control is passed to a function block 350. Otherwise, control is passed to a decision block 355.
The function block 350 sets the boundary strength (bS) equal to one, and passes control to the function block 365.
The decision block 355 determines whether or not the difference of the motion vectors of the pixels in two boundary sides is larger than one. If so, then control is passed to the function block 350. Otherwise, control is passed to a function block 360.
The function block 360 sets the boundary strength (bS) equal to zero, and passes control to the function block 365.
Hence, the boundary strength parameter (BS) is chosen according to the rules illustrated in FIG. 3. The result of applying these rules is that the filtering strength depends on the encoded data such that it is stronger at places where there is likely to be significant blocky artifacts (e.g., the boundary of an intra coded macroblock or a boundary between blocks that include coded coefficients). Based on the selection of boundary strength, the final filtering process is determined by the quantization parameter and the gradient of image samples across the boundary.
In FIG. 3, we can see that the MPEG-4 AVC Standard uses a higher boundary strength value for intra coded blocks, since the local intra prediction techniques used by the MPEG-4 AVC Standard use very simple models. However, these very simply models are unable to predict the whole set of components of the signal, resulting in more residue information for subsequent coding. The more prediction residue that is left for coding, the higher the probability of information loss during quantization which, in turn, may cause more blocky artifacts around the block boundary. However, if more advanced intra prediction approaches are introduced, such as non-local intra prediction techniques, the existing deblocking scheme is no longer suitable for those blocks.
Non-Local Intra Prediction
Non-local intra prediction accounts for those techniques that can exploit non-local information within the encoded picture in order to generate a prediction of the current coding block or region. Non-local information includes decoded data available at both the encoder and decoder. We can classify the non-local intra prediction techniques into forward prediction and backward prediction techniques based on the necessity of overhead transmitted. Displaced intra prediction and template matching prediction are typical forward and backward non-local intra prediction techniques, respectively.
Displaced Intra Prediction (DIP)
Motivated by inter motion compensation, displaced intra prediction reuses block patches in the reconstructed decoded area of a picture in order to predict the current block. Displaced intra prediction looks for the most similar block to the current block to be encoded within the reconstructed area of the picture that includes the current block. An intra displacement vector per block or partition is thus sent to the decoder as overhead. Turning to FIG. 4, an example of displaced intra prediction (DIP) is indicated generally by the reference numeral 400. The displaced intra prediction 400 involves a region to be encoded 410, a current block 411 to be encoded that is located within the region 410, a reconstructed region 430 located in the same picture as the patch 410, a candidate block 431 in the reconstructed region 410, and an intra displacement vector 440. Displaced intra prediction is very suitable for coding pictures with a lot of repetitive texture or structure patterns.
Template Matching Prediction (TMP)
Turning to FIG. 5, an example of template matching prediction (TMP) is indicated generally by the reference numeral 500. The template matching prediction 500 involves a region to be encoded 510, a current template 511 including a current block 512 to be encoded, a reconstructed region 530 in the same picture as the patch 510, a candidate template 533 within the reconstructed region 530, and a candidate block 531 within the candidate template 533.
As shown in FIG. 5, template matching prediction also generates predictions by reusing available reconstructed data at the encoder or decoder. Unlike displaced intra prediction, template matching prediction uses backward-adaptive texture synthesis techniques, requiring no overhead to be sent. Template matching prediction measures the similarity between the surrounding neighboring pixels (available at both the encoder and decoder) and candidate template for prediction rather than the original block data as used in displaced intra prediction. Since no additional overhead is required by template matching prediction, the target block can be partitioned into smaller blocks for prediction. This allows for more accurate modeling of high frequency components and complicated structures, reducing the residue to be coded. Template matching imposes a certain smoothness on the predicted block with respect to the neighboring blocks that have been used for the matching reference. This imposes some continuity in the prediction, which reduces the blocking effect due to prediction.
Non-Local Infra Prediction with IC
The previously described non-local intra prediction techniques typically use the Sum of Absolute Differences (SAD) or Sum Squared Errors (SSE) to measure the similarity between two templates (in template matching prediction) or two blocks (in displaced intra prediction). Although such measurement works well in most cases, it is not efficient enough when there is some mismatch between the templates or blocks. This may happen in the presence of illumination disparities or geometric variations, leading to sub-optimal prediction synthesis and a larger residue. This is due to the fact that non-local prediction techniques cannot always capture the local features of an image, such as contrast and brightness. Various adaptive illumination compensation (IC) approaches have been proposed to solve this problem explicitly or implicitly. Introducing illumination compensation to intra prediction results in further improvement in the prediction efficiency, which leads to a smaller amount of residue coefficients to be coded compared to the case without illumination compensation.
Non-local prediction techniques are able to provide better intra prediction. This reduces the amount of residue and, consequently, the probability of having blocking artifacts. Hence, the different nature of these prediction techniques requires the use of an adapted deblocking procedure.